{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2054630c",
   "metadata": {},
   "source": [
    "<font color='blue' size=5><b>Комментарий ревьюера</b></font>\n",
    "\n",
    "<font color='blue'>Привет, Александр! Меня зовут Павел Григорьев, и я буду проверять этот проект.<br>Моя основная цель - не указать на совершённые тобой ошибки, а поделиться своим опытом и помочь тебе совершенствоваться как профессионалу.<br>Спасибо за проделанную работу! Предлагаю общаться на «ты».</font>\n",
    "<details>\n",
    "\t<summary><u>Инструкция по организационным моментам (кликабельно)</u></summary>\n",
    "<font color='blue'>Я буду использовать различные цвета, чтобы было удобнее воспринимать мои комментарии:</font>\n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "<font color='blue'>синий текст - просто текст комментария</font>\n",
    "\n",
    "<font color='green'>✔️ и зеленый текст - все отлично</font>\n",
    "\n",
    "<font color='orange'>⚠️ и оранжевый текст - сделано все правильно, однако есть рекомендации, на что стоит обратить внимание</font>\n",
    "\n",
    "<font color='red'>❌ и красный текст - есть недочеты</font>\n",
    "\n",
    "\n",
    "</details>    \n",
    "    </br>\n",
    "<font color='blue'>Пожалуйста, не удаляй мои комментарии в случае возврата работы, так будет проще разобраться, какие были недочеты, а также сразу увидеть исправленное. </font>\n",
    "\n",
    "Ответы на мои комментарии лучше тоже помечать.\n",
    "Например: <font color='purple'><b>Комментарий студента</b></font>\n",
    "\n",
    "<font color='blue'><b>Давай смотреть, что получилось!</b></font>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "GEqQaHzsZ_bA",
   "metadata": {
    "id": "GEqQaHzsZ_bA"
   },
   "source": [
    "## Предсказание стоимости жилья\n",
    "\n",
    "В проекте вам нужно обучить модель линейной регрессии на данных о жилье в Калифорнии в 1990 году. На основе данных нужно предсказать медианную стоимость дома в жилом массиве. Обучите модель и сделайте предсказания на тестовой выборке. Для оценки качества модели используйте метрики RMSE, MAE и R2."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53871372",
   "metadata": {
    "id": "53871372"
   },
   "source": [
    "# Подготовка данных"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "df92e3ec",
   "metadata": {
    "id": "df92e3ec"
   },
   "outputs": [],
   "source": [
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "import pyspark\n",
    "from pyspark.sql import SparkSession\n",
    "from pyspark.sql.types import *\n",
    "import pyspark.sql.functions as F\n",
    "\n",
    "from pyspark.ml.feature import StringIndexer, VectorAssembler, StandardScaler\n",
    "from pyspark.ml.regression import LinearRegression\n",
    "from pyspark.ml.evaluation import RegressionEvaluator\n",
    "\n",
    "pyspark_version = pyspark.__version__\n",
    "if int(pyspark_version[:1]) == 3:\n",
    "    from pyspark.ml.feature import OneHotEncoder    \n",
    "elif int(pyspark_version[:1]) == 2:\n",
    "    from pyspark.ml.feature import OneHotEncodeEstimator"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b28946f9",
   "metadata": {},
   "source": [
    "<font color='blue'><b>Комментарий ревьюера: </b></font> ✔️\\\n",
    "<font color='green'> Хорошее оформление импортов! \\\n",
    "Импорты собраны в одной ячейке, разделены на функциональные группы пустой строкой.</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78863ba6",
   "metadata": {},
   "source": [
    "<font color='blue'><b>Комментарий ревьюера: </b></font> ✔️\\\n",
    "<font color='green'>Здорово, что есть условие по версии pyspark. \\\n",
    "Можно ещё сразу положить Энкодер в переменную, чтобы далее обращаться только к ней. </font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "98c71fb3",
   "metadata": {},
   "outputs": [],
   "source": [
    "RANDOM_SEED = 12321"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "7653ec3e",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "spark = (SparkSession.builder\n",
    "                    .master(\"local\")\n",
    "                    .appName(\"Housing - Logistic regression\")\n",
    "                    .getOrCreate())\n",
    "\n",
    "df = spark.read.option('header', 'true').csv('/datasets/housing.csv', inferSchema = True) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "fd617eab",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "longitude 0\n",
      "latitude 0\n",
      "housing_median_age 0\n",
      "total_rooms 0\n",
      "total_bedrooms 207\n",
      "population 0\n",
      "households 0\n",
      "median_income 0\n",
      "median_house_value 0\n",
      "ocean_proximity 20640\n"
     ]
    }
   ],
   "source": [
    "for column in df.columns:\n",
    "    check_col = F.col(column).cast(FloatType()).isNull()\n",
    "    print(column, df.filter(check_col).count())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb56a80d",
   "metadata": {},
   "source": [
    "<font color='blue'><b>Комментарий ревьюера: </b></font> ⚠️\\\n",
    "<font color='darkorange'> Нет пропусков?</font>\n",
    "\n",
    "<font color='purple'><b>Комментарий студента</b></font>\\\n",
    "<font color=#4B0082><b>Вот вот же они)</b></font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "d5ca53a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.na.fill(535, ['total_bedrooms'])\n",
    "df = df.na.fill('Unknown', ['ocean_proximity'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "69e8030e",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>summary</th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>housing_median_age</th>\n",
       "      <th>total_rooms</th>\n",
       "      <th>total_bedrooms</th>\n",
       "      <th>population</th>\n",
       "      <th>households</th>\n",
       "      <th>median_income</th>\n",
       "      <th>median_house_value</th>\n",
       "      <th>ocean_proximity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>count</td>\n",
       "      <td>20640</td>\n",
       "      <td>20640</td>\n",
       "      <td>20640</td>\n",
       "      <td>20640</td>\n",
       "      <td>20640</td>\n",
       "      <td>20640</td>\n",
       "      <td>20640</td>\n",
       "      <td>20640</td>\n",
       "      <td>20640</td>\n",
       "      <td>20640</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mean</td>\n",
       "      <td>-119.56970445736148</td>\n",
       "      <td>35.6318614341087</td>\n",
       "      <td>28.639486434108527</td>\n",
       "      <td>2635.7630813953488</td>\n",
       "      <td>537.8417635658915</td>\n",
       "      <td>1425.4767441860465</td>\n",
       "      <td>499.5396802325581</td>\n",
       "      <td>3.8706710029070246</td>\n",
       "      <td>206855.81690891474</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>stddev</td>\n",
       "      <td>2.003531723502584</td>\n",
       "      <td>2.135952397457101</td>\n",
       "      <td>12.58555761211163</td>\n",
       "      <td>2181.6152515827944</td>\n",
       "      <td>419.2666898951927</td>\n",
       "      <td>1132.46212176534</td>\n",
       "      <td>382.3297528316098</td>\n",
       "      <td>1.899821717945263</td>\n",
       "      <td>115395.61587441359</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>min</td>\n",
       "      <td>-124.35</td>\n",
       "      <td>32.54</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.4999</td>\n",
       "      <td>14999.0</td>\n",
       "      <td>&lt;1H OCEAN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>max</td>\n",
       "      <td>-114.31</td>\n",
       "      <td>41.95</td>\n",
       "      <td>52.0</td>\n",
       "      <td>39320.0</td>\n",
       "      <td>6445.0</td>\n",
       "      <td>35682.0</td>\n",
       "      <td>6082.0</td>\n",
       "      <td>15.0001</td>\n",
       "      <td>500001.0</td>\n",
       "      <td>NEAR OCEAN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  summary            longitude           latitude  housing_median_age  \\\n",
       "0   count                20640              20640               20640   \n",
       "1    mean  -119.56970445736148   35.6318614341087  28.639486434108527   \n",
       "2  stddev    2.003531723502584  2.135952397457101   12.58555761211163   \n",
       "3     min              -124.35              32.54                 1.0   \n",
       "4     max              -114.31              41.95                52.0   \n",
       "\n",
       "          total_rooms     total_bedrooms          population  \\\n",
       "0               20640              20640               20640   \n",
       "1  2635.7630813953488  537.8417635658915  1425.4767441860465   \n",
       "2  2181.6152515827944  419.2666898951927    1132.46212176534   \n",
       "3                 2.0                1.0                 3.0   \n",
       "4             39320.0             6445.0             35682.0   \n",
       "\n",
       "          households       median_income  median_house_value ocean_proximity  \n",
       "0              20640               20640               20640           20640  \n",
       "1  499.5396802325581  3.8706710029070246  206855.81690891474            None  \n",
       "2  382.3297528316098   1.899821717945263  115395.61587441359            None  \n",
       "3                1.0              0.4999             14999.0       <1H OCEAN  \n",
       "4             6082.0             15.0001            500001.0      NEAR OCEAN  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe().toPandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0d30915",
   "metadata": {},
   "source": [
    "<font color='blue'><b>Комментарий ревьюера: </b></font> ❌\\\n",
    "<font color='red'>По заданию пропуски нужно именно заполнить, выбрав значения по своему усмотрению.</font>\n",
    "\n",
    "<font color='purple'><b>Комментарий студента</b></font>\\\n",
    "<font color=#4B0082><b>Заполнил пропуски</b></font>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64e488ff",
   "metadata": {},
   "source": [
    "<font color='blue'><b>Комментарий ревьюера: </b></font> ❌\\\n",
    "<font color='red'>При использовании .toPandas() ты выгружаешь данные в оперативную память, что при работе с действительно большими данными приведет к переполнению памяти, этот метод применяется только к агрегированным таблицам(либо к ограниченным подвыборкам). Попробуй реализовать схожий функционал с применением методов pySpark.\n",
    "</font>\n",
    "\n",
    "<font color='purple'><b>Комментарий студента</b></font>\\\n",
    "<font color=#4B0082><b>Починил</b></font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "bf4cc42d",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('longitude', 'double'),\n",
       " ('latitude', 'double'),\n",
       " ('housing_median_age', 'double'),\n",
       " ('total_rooms', 'double'),\n",
       " ('total_bedrooms', 'double'),\n",
       " ('population', 'double'),\n",
       " ('households', 'double'),\n",
       " ('median_income', 'double'),\n",
       " ('median_house_value', 'double'),\n",
       " ('ocean_proximity', 'string')]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24f5022b",
   "metadata": {},
   "source": [
    "<font color='blue'><b>Комментарий ревьюера: </b></font> ✔️\\\n",
    "<font color='green'> Отлично!  Выведены типы данных.</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "2bf7f0c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.withColumn('longitude', F.col('longitude').cast(FloatType()))\n",
    "df = df.withColumn('latitude', F.col('latitude').cast(FloatType()))\n",
    "df = df.withColumn('housing_median_age', F.col('housing_median_age').cast(IntegerType()))\n",
    "df = df.withColumn('total_rooms', F.col('total_rooms').cast(IntegerType()))\n",
    "df = df.withColumn('total_bedrooms', F.col('total_bedrooms').cast(IntegerType()))\n",
    "df = df.withColumn('population', F.col('population').cast(IntegerType()))\n",
    "df = df.withColumn('households', F.col('households').cast(IntegerType()))\n",
    "df = df.withColumn('median_income', F.col('median_income').cast(FloatType()))\n",
    "df = df.withColumn('median_house_value', F.col('median_house_value').cast(FloatType()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "309fc817",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('longitude', 'float'),\n",
       " ('latitude', 'float'),\n",
       " ('housing_median_age', 'int'),\n",
       " ('total_rooms', 'int'),\n",
       " ('total_bedrooms', 'int'),\n",
       " ('population', 'int'),\n",
       " ('households', 'int'),\n",
       " ('median_income', 'float'),\n",
       " ('median_house_value', 'float'),\n",
       " ('ocean_proximity', 'string')]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "60dc4d78",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "16515 4125\n"
     ]
    }
   ],
   "source": [
    "# разделим на тест и трейн\n",
    "train_data, test_data = df.randomSplit([.8,.2], seed=RANDOM_SEED)\n",
    "print(train_data.count(), test_data.count()) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "adc0c5b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "categorical_cols = ['ocean_proximity']\n",
    "numerical_cols  = [\"longitude\", \"latitude\", \"housing_median_age\", \n",
    "                   \"total_rooms\", \"total_bedrooms\", \"population\", \n",
    "                   \"households\", \"median_income\"]\n",
    "target = \"median_house_value\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "e115c1b7",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------------+\n",
      "|ocean_proximity|\n",
      "+---------------+\n",
      "|     NEAR OCEAN|\n",
      "|     NEAR OCEAN|\n",
      "|     NEAR OCEAN|\n",
      "+---------------+\n",
      "only showing top 3 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "indexer = StringIndexer(inputCols=categorical_cols, \n",
    "                        outputCols=[c+'_idx' for c in categorical_cols]) \n",
    "fitter = indexer.fit(train_data)\n",
    "train_data = fitter.transform(train_data)\n",
    "test_data = fitter.transform(test_data)\n",
    "\n",
    "cols = [c for c in df.columns for i in categorical_cols if (c.startswith(i))]\n",
    "train_data.select(cols).show(3) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e164deff",
   "metadata": {},
   "source": [
    "<font color='blue'><b>Комментарий ревьюера: </b></font> ❌\\\n",
    "<font color='red'>Кодирование и Стандартизацию нужно проводить после разделения на выборки трансформерами обучеными на обучающих данных.</font>\n",
    "\n",
    "<font color='purple'><b>Комментарий студента</b></font>\\\n",
    "<font color=#4B0082><b>Сделал. Перенёс разделение выше и выполнил трансформации через данные трейна.</b></font>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afbd88a8",
   "metadata": {},
   "source": [
    "<font color='blue'><b>Комментарий ревьюера: </b></font> ⚠️\\\n",
    "<font color='blue'>Можно использовать [Pipeline](https://habr.com/ru/post/522296/)</font>\n",
    "\n",
    "<font color='purple'><b>Комментарий студента</b></font>\\\n",
    "<font color=#4B0082><b>Да, всегда использую пайплайн в ml проектах. Но в теории пайпалайном, почему то, назвали простую последовательность преобразований. Статью прочёл, она информативная и с примерами - буду использовать.</b></font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "64282c57",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------------+\n",
      "|ocean_proximity|\n",
      "+---------------+\n",
      "|     NEAR OCEAN|\n",
      "|     NEAR OCEAN|\n",
      "|     NEAR OCEAN|\n",
      "+---------------+\n",
      "only showing top 3 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "encoder = OneHotEncoder(inputCols=[c+'_idx' for c in categorical_cols],\n",
    "                        outputCols=[c+'_ohe' for c in categorical_cols])\n",
    "fitter = encoder.fit(train_data)\n",
    "train_data = fitter.transform(train_data)\n",
    "test_data = fitter.transform(test_data)\n",
    "\n",
    "cols = [c for c in df.columns for i in categorical_cols if (c.startswith(i))]\n",
    "train_data.select(cols).show(3) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "186d1870",
   "metadata": {},
   "source": [
    "<font color='blue'><b>Комментарий ревьюера: </b></font> ✔️ <br>\n",
    "<font color='green'>Верно, что используешь OHE.</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "5d424ba0",
   "metadata": {},
   "outputs": [],
   "source": [
    "categorical_assembler = \\\n",
    "        VectorAssembler(inputCols=[c+'_ohe' for c in categorical_cols],\n",
    "                                        outputCol=\"categorical_features\")\n",
    "train_data = categorical_assembler.transform(train_data)\n",
    "test_data = categorical_assembler.transform(test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "05013a4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "numerical_assembler = VectorAssembler(inputCols=numerical_cols, outputCol=\"numerical_features\")\n",
    "train_data = numerical_assembler.transform(train_data)\n",
    "test_data = numerical_assembler.transform(test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "11929f4b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "standardScaler = StandardScaler(inputCol='numerical_features', outputCol=\"numerical_features_scaled\")\n",
    "fitter = standardScaler.fit(train_data)\n",
    "train_data = fitter.transform(train_data)\n",
    "test_data = fitter.transform(test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "3711edad",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['longitude', 'latitude', 'housing_median_age', 'total_rooms', 'total_bedrooms', 'population', 'households', 'median_income', 'median_house_value', 'ocean_proximity']\n"
     ]
    }
   ],
   "source": [
    "print(df.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "47c97507",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+-------------------------+\n",
      "|categorical_features|numerical_features_scaled|\n",
      "+--------------------+-------------------------+\n",
      "|       (4,[2],[1.0])|     [-62.087394450650...|\n",
      "|       (4,[2],[1.0])|     [-62.062431961328...|\n",
      "|       (4,[2],[1.0])|     [-62.062431961328...|\n",
      "+--------------------+-------------------------+\n",
      "only showing top 3 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "all_features = ['categorical_features','numerical_features_scaled']\n",
    "\n",
    "final_assembler = VectorAssembler(inputCols=all_features, \n",
    "                                  outputCol=\"features\") \n",
    "train_data = final_assembler.transform(train_data)\n",
    "test_data = final_assembler.transform(test_data)\n",
    "\n",
    "train_data.select(all_features).show(3) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f0a82f3",
   "metadata": {
    "id": "9f0a82f3"
   },
   "source": [
    "# Обучение моделей"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa4358ab",
   "metadata": {},
   "source": [
    "<font color='blue'><b>Комментарий ревьюера: </b></font> ✔️\\\n",
    "<font color='green'> Здорово , что у нас есть выборка для тестов!</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "265227d5",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "23/07/12 12:06:20 WARN Instrumentation: [ed8bd6d4] regParam is zero, which might cause numerical instability and overfitting.\n",
      "23/07/12 12:06:20 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS\n",
      "23/07/12 12:06:20 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS\n",
      "23/07/12 12:06:21 WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK\n",
      "23/07/12 12:06:21 WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeRefLAPACK\n"
     ]
    }
   ],
   "source": [
    "lr1 = LinearRegression(labelCol=target, featuresCol='features')\n",
    "\n",
    "model1 = lr1.fit(train_data) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "2778ef94",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------+------------------+\n",
      "|median_house_value|        prediction|\n",
      "+------------------+------------------+\n",
      "|           79000.0|184784.96457589697|\n",
      "|           70500.0| 145578.4707786031|\n",
      "|           64600.0| 134252.9767572144|\n",
      "|           81300.0|153131.19892569445|\n",
      "|           75500.0| 160162.9521834054|\n",
      "|           85100.0| 180208.6472232798|\n",
      "|           85600.0| 189570.5222724015|\n",
      "|           82000.0|201416.83746790187|\n",
      "|           90600.0|  181322.250163408|\n",
      "|           74100.0| 150687.3736915365|\n",
      "|           69500.0|114835.50782892341|\n",
      "|           90000.0|210354.13294354128|\n",
      "|           86400.0|194381.71901026508|\n",
      "|           76800.0|149416.36199384276|\n",
      "|          128100.0|222344.05520341126|\n",
      "|           92800.0| 210969.6577821509|\n",
      "|           94800.0| 226143.3820285434|\n",
      "|           86900.0| 153577.4737201701|\n",
      "|           92600.0|170699.81702103047|\n",
      "|           78400.0|124357.68216068391|\n",
      "+------------------+------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "predictions1 = model1.transform(test_data)\n",
    "\n",
    "predictedLabes1 = predictions1.select(target, \"prediction\")\n",
    "predictedLabes1.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "828ad5f3",
   "metadata": {},
   "source": [
    "<font color='blue'><b>Комментарий ревьюера : </b></font> ✔️\\\n",
    "<font color='green'> 👍</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "f44eadba",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "23/07/12 12:06:23 WARN Instrumentation: [05c52006] regParam is zero, which might cause numerical instability and overfitting.\n"
     ]
    }
   ],
   "source": [
    "lr2 = LinearRegression(labelCol=target, featuresCol='numerical_features_scaled')\n",
    "\n",
    "model2 = lr2.fit(train_data) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "1c5c2f83",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------+------------------+\n",
      "|median_house_value|        prediction|\n",
      "+------------------+------------------+\n",
      "|           79000.0| 155887.3883871357|\n",
      "|           70500.0|112091.12482968485|\n",
      "|           64600.0|100967.52679055277|\n",
      "|           81300.0|119406.22399751004|\n",
      "|           75500.0|128645.12284473097|\n",
      "|           85100.0|150878.96364618186|\n",
      "|           85600.0|157183.22953275312|\n",
      "|           82000.0|169627.07351799076|\n",
      "|           90600.0|148888.61146485852|\n",
      "|           74100.0|119243.48233028268|\n",
      "|           69500.0|62517.699619129766|\n",
      "|           90000.0| 177745.4250109787|\n",
      "|           86400.0|161243.12849511765|\n",
      "|           76800.0|117401.79133779695|\n",
      "|          128100.0|192613.94207159244|\n",
      "|           92800.0| 178012.4575115433|\n",
      "|           94800.0|207341.23763665278|\n",
      "|           86900.0|119359.37586823804|\n",
      "|           92600.0| 133119.0467339009|\n",
      "|           78400.0| 85214.49613918317|\n",
      "+------------------+------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "predictions2 = model2.transform(test_data)\n",
    "\n",
    "predictedLabes2 = predictions2.select(target, \"prediction\")\n",
    "predictedLabes2.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9d9ba25",
   "metadata": {},
   "source": [
    "<font color='blue'><b>Комментарий ревьюера : </b></font> ✔️\\\n",
    "<font color='green'> 👍</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "757fd103",
   "metadata": {
    "id": "757fd103"
   },
   "source": [
    "# Анализ результатов"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "b71c523b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE модели для целого датасета:  68515.87755589013\n",
      "MAE модели для целого датасета:  50021.948676277134\n",
      "R2 модели для целого датасета:  0.6454595655138406\n"
     ]
    }
   ],
   "source": [
    "rmse1  = RegressionEvaluator(\n",
    "    labelCol=target, metricName='rmse').evaluate(predictions1)\n",
    "mae1 = RegressionEvaluator(\n",
    "    labelCol=target, metricName='mae').evaluate(predictions1)\n",
    "r21 = RegressionEvaluator(\n",
    "    labelCol=target, metricName='r2').evaluate(predictions1)\n",
    "\n",
    "print('RMSE модели для целого датасета: ', rmse1)\n",
    "print('MAE модели для целого датасета: ', mae1) \n",
    "print('R2 модели для целого датасета: ', r21) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "080b5a86",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE модели без категориальных параметров:  69145.41652885452\n",
      "MAE модели без категориальных параметров:  50895.57364243959\n",
      "R2 модели без категориальных параметров:  0.6389144428155036\n"
     ]
    }
   ],
   "source": [
    "rmse2  = RegressionEvaluator(\n",
    "    labelCol=target, metricName='rmse').evaluate(predictions2)\n",
    "mae2 = RegressionEvaluator(\n",
    "    labelCol=target, metricName='mae').evaluate(predictions2)\n",
    "r22 = RegressionEvaluator(\n",
    "    labelCol=target, metricName='r2').evaluate(predictions2)\n",
    "\n",
    "print('RMSE модели без категориальных параметров: ', rmse2)\n",
    "print('MAE модели без категориальных параметров: ', mae2) \n",
    "print('R2 модели без категориальных параметров: ', r22) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4540c1e5",
   "metadata": {},
   "source": [
    "<font color='blue'><b>Комментарий ревьюера :</b></font> ✔️\\\n",
    "<font color='green'>Тестирование проведено верно.</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1dc3315c",
   "metadata": {},
   "source": [
    "ВЫВОД: Линейный модели показывают низкую точность в данной задаче. Категориальный столбец немного портит результат для линейной модели. В данной задаче требуется уделить больше времени на предобработку данных и обучение более тяжеловесных моделей."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "af860801",
   "metadata": {},
   "outputs": [],
   "source": [
    "spark.stop()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a079bc99",
   "metadata": {},
   "source": [
    "<font color='blue'><b>Комментарий ревьюера: </b></font> ✔️\\\n",
    "<font color='green'> Очень приятно видеть вывод в конце проекта!\\\n",
    "Приведены ответы на главные вопросы проекта.</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "708401ab",
   "metadata": {},
   "source": [
    "<font color='blue'><b>Итоговый комментарий ревьюера</b></font>\\\n",
    "<font color='green'> Александр, хороший проект получился!\n",
    "Большое спасибо за проделанную работу. Видно, что приложено много усилий.\n",
    "</font>\n",
    "\n",
    "<font color='blue'>Что нужно исправить:</font>\n",
    "<ul><font color='red'>Поправь работу с данными (toPandas только для агрегированных данных).</font></ul>\n",
    "<ul><font color='red'>Заполни пропуски, выбрав значения по своему усмотрению.</font></ul>\n",
    "<ul><font color='red'>Стандартизация и Кодировка для train и test по отдельности обработчиком обученным на train.</font></ul>\n",
    "\n",
    "<font color='darkorange'>Не забывай останавливать сессию `spark.stop()`</font>\n",
    "\n",
    "<font color='blue'><b>Жду новую версию проекта :)</b></font>"
   ]
  }
 ],
 "metadata": {
  "ExecuteTimeLog": [
   {
    "duration": 775,
    "start_time": "2023-07-07T09:10:32.887Z"
   },
   {
    "duration": 4,
    "start_time": "2023-07-07T09:10:33.664Z"
   },
   {
    "duration": 17320,
    "start_time": "2023-07-07T09:10:33.669Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T09:10:50.991Z"
   },
   {
    "duration": 157,
    "start_time": "2023-07-07T09:11:09.282Z"
   },
   {
    "duration": 9537,
    "start_time": "2023-07-07T09:11:22.282Z"
   },
   {
    "duration": 4888,
    "start_time": "2023-07-07T09:11:31.821Z"
   },
   {
    "duration": 2087,
    "start_time": "2023-07-07T09:11:50.852Z"
   },
   {
    "duration": 4,
    "start_time": "2023-07-07T09:14:37.346Z"
   },
   {
    "duration": 2481,
    "start_time": "2023-07-07T09:14:38.853Z"
   },
   {
    "duration": 701,
    "start_time": "2023-07-07T09:14:57.644Z"
   },
   {
    "duration": 57,
    "start_time": "2023-07-07T09:15:19.600Z"
   },
   {
    "duration": 6,
    "start_time": "2023-07-07T09:17:41.271Z"
   },
   {
    "duration": 193,
    "start_time": "2023-07-07T09:17:50.932Z"
   },
   {
    "duration": 133,
    "start_time": "2023-07-07T09:18:12.458Z"
   },
   {
    "duration": 5,
    "start_time": "2023-07-07T09:18:40.517Z"
   },
   {
    "duration": 23,
    "start_time": "2023-07-07T09:18:40.525Z"
   },
   {
    "duration": 698,
    "start_time": "2023-07-07T09:18:40.550Z"
   },
   {
    "duration": 1395,
    "start_time": "2023-07-07T09:18:41.250Z"
   },
   {
    "duration": 63,
    "start_time": "2023-07-07T09:18:42.648Z"
   },
   {
    "duration": 935,
    "start_time": "2023-07-07T09:18:42.713Z"
   },
   {
    "duration": 366,
    "start_time": "2023-07-07T09:18:43.650Z"
   },
   {
    "duration": 26,
    "start_time": "2023-07-07T09:18:44.018Z"
   },
   {
    "duration": 279,
    "start_time": "2023-07-07T09:18:44.109Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T09:18:44.391Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T09:18:44.391Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T09:18:44.393Z"
   },
   {
    "duration": 4,
    "start_time": "2023-07-07T09:20:05.797Z"
   },
   {
    "duration": 30,
    "start_time": "2023-07-07T09:20:08.872Z"
   },
   {
    "duration": 792,
    "start_time": "2023-07-07T09:20:11.371Z"
   },
   {
    "duration": 527,
    "start_time": "2023-07-07T09:20:22.827Z"
   },
   {
    "duration": 4,
    "start_time": "2023-07-07T09:20:30.529Z"
   },
   {
    "duration": 106,
    "start_time": "2023-07-07T09:20:30.536Z"
   },
   {
    "duration": 992,
    "start_time": "2023-07-07T09:20:30.644Z"
   },
   {
    "duration": 1285,
    "start_time": "2023-07-07T09:20:31.637Z"
   },
   {
    "duration": 4,
    "start_time": "2023-07-07T09:20:32.926Z"
   },
   {
    "duration": 879,
    "start_time": "2023-07-07T09:20:32.933Z"
   },
   {
    "duration": 335,
    "start_time": "2023-07-07T09:20:33.816Z"
   },
   {
    "duration": 77,
    "start_time": "2023-07-07T09:20:34.153Z"
   },
   {
    "duration": 101,
    "start_time": "2023-07-07T09:20:34.232Z"
   },
   {
    "duration": 624,
    "start_time": "2023-07-07T09:20:34.335Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T09:20:34.961Z"
   },
   {
    "duration": 7,
    "start_time": "2023-07-07T09:20:37.487Z"
   },
   {
    "duration": 5,
    "start_time": "2023-07-07T09:22:28.119Z"
   },
   {
    "duration": 16,
    "start_time": "2023-07-07T09:22:28.127Z"
   },
   {
    "duration": 570,
    "start_time": "2023-07-07T09:22:28.146Z"
   },
   {
    "duration": 1314,
    "start_time": "2023-07-07T09:22:28.719Z"
   },
   {
    "duration": 4,
    "start_time": "2023-07-07T09:22:30.035Z"
   },
   {
    "duration": 1016,
    "start_time": "2023-07-07T09:22:30.041Z"
   },
   {
    "duration": 385,
    "start_time": "2023-07-07T09:22:31.110Z"
   },
   {
    "duration": 34,
    "start_time": "2023-07-07T09:22:31.497Z"
   },
   {
    "duration": 26,
    "start_time": "2023-07-07T09:22:31.534Z"
   },
   {
    "duration": 1058,
    "start_time": "2023-07-07T09:22:31.562Z"
   },
   {
    "duration": 8,
    "start_time": "2023-07-07T09:22:32.622Z"
   },
   {
    "duration": 365,
    "start_time": "2023-07-07T09:22:53.064Z"
   },
   {
    "duration": 3377,
    "start_time": "2023-07-07T09:23:32.034Z"
   },
   {
    "duration": 7850425,
    "start_time": "2023-07-07T09:23:38.640Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T11:34:29.114Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T11:34:29.115Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T11:34:29.116Z"
   },
   {
    "duration": 9,
    "start_time": "2023-07-07T11:39:22.886Z"
   },
   {
    "duration": 56,
    "start_time": "2023-07-07T11:39:31.085Z"
   },
   {
    "duration": 19,
    "start_time": "2023-07-07T11:39:38.368Z"
   },
   {
    "duration": 5,
    "start_time": "2023-07-07T11:40:21.485Z"
   },
   {
    "duration": 27,
    "start_time": "2023-07-07T11:40:21.494Z"
   },
   {
    "duration": 43,
    "start_time": "2023-07-07T11:40:21.527Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T11:40:21.573Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T11:40:21.575Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T11:40:21.577Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T11:40:21.579Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T11:40:21.581Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T11:40:21.582Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T11:40:21.583Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T11:40:21.585Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T11:40:21.587Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T11:40:21.589Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T11:40:21.590Z"
   },
   {
    "duration": 12,
    "start_time": "2023-07-07T11:49:00.806Z"
   },
   {
    "duration": 251,
    "start_time": "2023-07-07T11:49:00.822Z"
   },
   {
    "duration": 196,
    "start_time": "2023-07-07T11:49:01.075Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T11:49:01.273Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T11:49:01.275Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T11:49:01.276Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T11:49:01.277Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T11:49:01.278Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T11:49:01.279Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T11:49:01.280Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T11:49:01.281Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T11:49:01.282Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T11:49:01.283Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T11:49:01.284Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T11:49:01.286Z"
   },
   {
    "duration": 1033,
    "start_time": "2023-07-07T11:49:18.209Z"
   },
   {
    "duration": 3,
    "start_time": "2023-07-07T11:49:19.245Z"
   },
   {
    "duration": 31197,
    "start_time": "2023-07-07T11:49:19.250Z"
   },
   {
    "duration": 7024,
    "start_time": "2023-07-07T11:49:50.510Z"
   },
   {
    "duration": 3,
    "start_time": "2023-07-07T11:49:57.537Z"
   },
   {
    "duration": 3392,
    "start_time": "2023-07-07T11:49:57.542Z"
   },
   {
    "duration": 1173,
    "start_time": "2023-07-07T11:50:00.936Z"
   },
   {
    "duration": 135,
    "start_time": "2023-07-07T11:50:02.111Z"
   },
   {
    "duration": 214,
    "start_time": "2023-07-07T11:50:02.248Z"
   },
   {
    "duration": 2171,
    "start_time": "2023-07-07T11:50:02.465Z"
   },
   {
    "duration": 6,
    "start_time": "2023-07-07T11:50:04.638Z"
   },
   {
    "duration": 718,
    "start_time": "2023-07-07T11:50:04.710Z"
   },
   {
    "duration": 4789,
    "start_time": "2023-07-07T11:50:05.431Z"
   },
   {
    "duration": 782,
    "start_time": "2023-07-07T13:01:56.280Z"
   },
   {
    "duration": 3,
    "start_time": "2023-07-07T13:01:58.706Z"
   },
   {
    "duration": 27439,
    "start_time": "2023-07-07T13:02:00.597Z"
   },
   {
    "duration": 5701,
    "start_time": "2023-07-07T13:02:28.038Z"
   },
   {
    "duration": 4,
    "start_time": "2023-07-07T13:02:33.740Z"
   },
   {
    "duration": 2868,
    "start_time": "2023-07-07T13:02:33.746Z"
   },
   {
    "duration": 1017,
    "start_time": "2023-07-07T13:02:36.616Z"
   },
   {
    "duration": 117,
    "start_time": "2023-07-07T13:02:37.635Z"
   },
   {
    "duration": 155,
    "start_time": "2023-07-07T13:02:37.754Z"
   },
   {
    "duration": 1636,
    "start_time": "2023-07-07T13:02:37.911Z"
   },
   {
    "duration": 65,
    "start_time": "2023-07-07T13:02:39.549Z"
   },
   {
    "duration": 607,
    "start_time": "2023-07-07T13:02:39.618Z"
   },
   {
    "duration": 4481,
    "start_time": "2023-07-07T13:02:40.228Z"
   },
   {
    "duration": 7923132,
    "start_time": "2023-07-07T13:02:44.711Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T15:14:47.845Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-07T15:14:47.847Z"
   },
   {
    "duration": 1,
    "start_time": "2023-07-07T15:14:47.848Z"
   },
   {
    "duration": 698,
    "start_time": "2023-07-11T09:00:18.548Z"
   },
   {
    "duration": 3,
    "start_time": "2023-07-11T09:00:19.248Z"
   },
   {
    "duration": 26662,
    "start_time": "2023-07-11T09:00:19.252Z"
   },
   {
    "duration": 5408,
    "start_time": "2023-07-11T09:00:45.918Z"
   },
   {
    "duration": 7,
    "start_time": "2023-07-11T09:00:51.328Z"
   },
   {
    "duration": 315,
    "start_time": "2023-07-11T09:07:08.101Z"
   },
   {
    "duration": 5,
    "start_time": "2023-07-11T09:07:13.870Z"
   },
   {
    "duration": 3,
    "start_time": "2023-07-11T09:07:21.776Z"
   },
   {
    "duration": 2307,
    "start_time": "2023-07-11T09:07:22.324Z"
   },
   {
    "duration": 686,
    "start_time": "2023-07-11T09:07:26.645Z"
   },
   {
    "duration": 83,
    "start_time": "2023-07-11T09:07:29.033Z"
   },
   {
    "duration": 45,
    "start_time": "2023-07-11T09:07:30.340Z"
   },
   {
    "duration": 1422,
    "start_time": "2023-07-11T09:07:31.697Z"
   },
   {
    "duration": 6,
    "start_time": "2023-07-11T09:07:36.800Z"
   },
   {
    "duration": 462,
    "start_time": "2023-07-11T09:07:39.050Z"
   },
   {
    "duration": 3601,
    "start_time": "2023-07-11T09:07:41.515Z"
   },
   {
    "duration": 7130130,
    "start_time": "2023-07-11T09:07:48.617Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:06:38.749Z"
   },
   {
    "duration": 9,
    "start_time": "2023-07-11T11:52:29.657Z"
   },
   {
    "duration": 11,
    "start_time": "2023-07-11T11:52:38.355Z"
   },
   {
    "duration": 9,
    "start_time": "2023-07-11T11:53:24.393Z"
   },
   {
    "duration": 6,
    "start_time": "2023-07-11T11:53:27.257Z"
   },
   {
    "duration": 26,
    "start_time": "2023-07-11T11:53:33.727Z"
   },
   {
    "duration": 4,
    "start_time": "2023-07-11T11:53:42.038Z"
   },
   {
    "duration": 7,
    "start_time": "2023-07-11T11:53:42.045Z"
   },
   {
    "duration": 55,
    "start_time": "2023-07-11T11:53:42.054Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:53:42.112Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:53:42.114Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:53:42.115Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:53:42.116Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:53:42.118Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:53:42.119Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:53:42.121Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:53:42.122Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:53:42.123Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:53:42.126Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:53:42.127Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:53:42.128Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:53:42.130Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:53:42.131Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:53:42.132Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:53:42.202Z"
   },
   {
    "duration": 1143,
    "start_time": "2023-07-11T11:54:19.879Z"
   },
   {
    "duration": 3,
    "start_time": "2023-07-11T11:54:21.024Z"
   },
   {
    "duration": 37491,
    "start_time": "2023-07-11T11:54:21.029Z"
   },
   {
    "duration": 1117,
    "start_time": "2023-07-11T11:54:58.522Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:54:59.642Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:54:59.644Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:54:59.701Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:54:59.703Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:54:59.704Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:54:59.706Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:54:59.707Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:54:59.708Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:54:59.709Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:54:59.710Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:54:59.713Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:54:59.714Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:54:59.715Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:54:59.716Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T11:54:59.718Z"
   },
   {
    "duration": 10922,
    "start_time": "2023-07-11T12:07:47.185Z"
   },
   {
    "duration": 6786,
    "start_time": "2023-07-11T12:07:59.319Z"
   },
   {
    "duration": 2302,
    "start_time": "2023-07-11T12:08:06.109Z"
   },
   {
    "duration": 6,
    "start_time": "2023-07-11T12:08:08.415Z"
   },
   {
    "duration": 506,
    "start_time": "2023-07-11T12:08:08.423Z"
   },
   {
    "duration": 10,
    "start_time": "2023-07-11T12:08:09.003Z"
   },
   {
    "duration": 9,
    "start_time": "2023-07-11T12:08:09.022Z"
   },
   {
    "duration": 4600,
    "start_time": "2023-07-11T12:08:09.032Z"
   },
   {
    "duration": 1887,
    "start_time": "2023-07-11T12:08:13.635Z"
   },
   {
    "duration": 296,
    "start_time": "2023-07-11T12:08:15.524Z"
   },
   {
    "duration": 184,
    "start_time": "2023-07-11T12:08:15.825Z"
   },
   {
    "duration": 2796,
    "start_time": "2023-07-11T12:08:16.010Z"
   },
   {
    "duration": 17,
    "start_time": "2023-07-11T12:08:18.809Z"
   },
   {
    "duration": 985,
    "start_time": "2023-07-11T12:08:18.828Z"
   },
   {
    "duration": 7201,
    "start_time": "2023-07-11T12:08:19.815Z"
   },
   {
    "duration": 8985,
    "start_time": "2023-07-11T12:08:27.017Z"
   },
   {
    "duration": 520,
    "start_time": "2023-07-11T12:08:36.008Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T12:08:36.532Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T12:08:36.534Z"
   },
   {
    "duration": 1969,
    "start_time": "2023-07-11T12:09:23.055Z"
   },
   {
    "duration": 2104,
    "start_time": "2023-07-11T12:10:01.918Z"
   },
   {
    "duration": 1194,
    "start_time": "2023-07-11T12:10:26.948Z"
   },
   {
    "duration": 3,
    "start_time": "2023-07-11T12:10:28.144Z"
   },
   {
    "duration": 40960,
    "start_time": "2023-07-11T12:10:28.148Z"
   },
   {
    "duration": 13421,
    "start_time": "2023-07-11T12:11:09.110Z"
   },
   {
    "duration": 3099,
    "start_time": "2023-07-11T12:11:22.533Z"
   },
   {
    "duration": 6,
    "start_time": "2023-07-11T12:11:25.635Z"
   },
   {
    "duration": 582,
    "start_time": "2023-07-11T12:11:25.645Z"
   },
   {
    "duration": 84,
    "start_time": "2023-07-11T12:11:26.230Z"
   },
   {
    "duration": 23,
    "start_time": "2023-07-11T12:11:26.320Z"
   },
   {
    "duration": 5185,
    "start_time": "2023-07-11T12:11:26.345Z"
   },
   {
    "duration": 1977,
    "start_time": "2023-07-11T12:11:31.532Z"
   },
   {
    "duration": 210,
    "start_time": "2023-07-11T12:11:33.510Z"
   },
   {
    "duration": 190,
    "start_time": "2023-07-11T12:11:33.722Z"
   },
   {
    "duration": 3184,
    "start_time": "2023-07-11T12:11:33.921Z"
   },
   {
    "duration": 24,
    "start_time": "2023-07-11T12:11:37.106Z"
   },
   {
    "duration": 1100,
    "start_time": "2023-07-11T12:11:37.133Z"
   },
   {
    "duration": 7487,
    "start_time": "2023-07-11T12:11:38.235Z"
   },
   {
    "duration": 8094,
    "start_time": "2023-07-11T12:11:45.723Z"
   },
   {
    "duration": 1907,
    "start_time": "2023-07-11T12:11:53.819Z"
   },
   {
    "duration": 635,
    "start_time": "2023-07-11T12:11:55.728Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-11T12:11:56.401Z"
   },
   {
    "duration": 7286,
    "start_time": "2023-07-11T12:17:47.643Z"
   },
   {
    "duration": 2013,
    "start_time": "2023-07-11T12:17:55.003Z"
   },
   {
    "duration": 10,
    "start_time": "2023-07-11T12:31:20.844Z"
   },
   {
    "duration": 5,
    "start_time": "2023-07-11T12:31:36.918Z"
   },
   {
    "duration": 19,
    "start_time": "2023-07-11T12:31:40.536Z"
   },
   {
    "duration": 1247,
    "start_time": "2023-07-11T12:32:01.694Z"
   },
   {
    "duration": 64,
    "start_time": "2023-07-11T12:32:02.944Z"
   },
   {
    "duration": 41118,
    "start_time": "2023-07-11T12:32:03.013Z"
   },
   {
    "duration": 12194,
    "start_time": "2023-07-11T12:32:44.133Z"
   },
   {
    "duration": 2318,
    "start_time": "2023-07-11T12:32:56.403Z"
   },
   {
    "duration": 4,
    "start_time": "2023-07-11T12:32:58.723Z"
   },
   {
    "duration": 575,
    "start_time": "2023-07-11T12:32:58.729Z"
   },
   {
    "duration": 12,
    "start_time": "2023-07-11T12:32:59.312Z"
   },
   {
    "duration": 78,
    "start_time": "2023-07-11T12:32:59.326Z"
   },
   {
    "duration": 4401,
    "start_time": "2023-07-11T12:32:59.409Z"
   },
   {
    "duration": 1603,
    "start_time": "2023-07-11T12:33:03.811Z"
   },
   {
    "duration": 109,
    "start_time": "2023-07-11T12:33:05.416Z"
   },
   {
    "duration": 190,
    "start_time": "2023-07-11T12:33:05.527Z"
   },
   {
    "duration": 2690,
    "start_time": "2023-07-11T12:33:05.720Z"
   },
   {
    "duration": 6,
    "start_time": "2023-07-11T12:33:08.411Z"
   },
   {
    "duration": 1101,
    "start_time": "2023-07-11T12:33:08.421Z"
   },
   {
    "duration": 6598,
    "start_time": "2023-07-11T12:33:09.524Z"
   },
   {
    "duration": 8195,
    "start_time": "2023-07-11T12:33:16.125Z"
   },
   {
    "duration": 2086,
    "start_time": "2023-07-11T12:33:24.323Z"
   },
   {
    "duration": 6006,
    "start_time": "2023-07-11T12:33:26.410Z"
   },
   {
    "duration": 2102,
    "start_time": "2023-07-11T12:33:32.421Z"
   },
   {
    "duration": 5203,
    "start_time": "2023-07-11T12:33:34.525Z"
   },
   {
    "duration": 4709,
    "start_time": "2023-07-11T12:33:39.804Z"
   },
   {
    "duration": 766,
    "start_time": "2023-07-11T15:21:13.590Z"
   },
   {
    "duration": 2,
    "start_time": "2023-07-11T15:21:14.358Z"
   },
   {
    "duration": 24338,
    "start_time": "2023-07-11T15:21:14.361Z"
   },
   {
    "duration": 6297,
    "start_time": "2023-07-11T15:21:38.700Z"
   },
   {
    "duration": 3440,
    "start_time": "2023-07-12T07:09:52.563Z"
   },
   {
    "duration": 5,
    "start_time": "2023-07-12T11:45:56.090Z"
   },
   {
    "duration": 8,
    "start_time": "2023-07-12T11:45:56.097Z"
   },
   {
    "duration": 705,
    "start_time": "2023-07-12T11:45:56.106Z"
   },
   {
    "duration": 2832,
    "start_time": "2023-07-12T11:45:56.813Z"
   },
   {
    "duration": 138,
    "start_time": "2023-07-12T11:45:59.647Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-12T11:45:59.786Z"
   },
   {
    "duration": 13,
    "start_time": "2023-07-12T11:47:41.104Z"
   },
   {
    "duration": 13,
    "start_time": "2023-07-12T11:47:45.840Z"
   },
   {
    "duration": 33,
    "start_time": "2023-07-12T11:48:47.820Z"
   },
   {
    "duration": 2388,
    "start_time": "2023-07-12T11:48:49.600Z"
   },
   {
    "duration": 6,
    "start_time": "2023-07-12T11:49:39.163Z"
   },
   {
    "duration": 125,
    "start_time": "2023-07-12T11:49:40.580Z"
   },
   {
    "duration": 5,
    "start_time": "2023-07-12T11:49:41.732Z"
   },
   {
    "duration": 1594,
    "start_time": "2023-07-12T11:49:43.284Z"
   },
   {
    "duration": 3,
    "start_time": "2023-07-12T11:50:41.020Z"
   },
   {
    "duration": 1588,
    "start_time": "2023-07-12T11:50:42.011Z"
   },
   {
    "duration": 188,
    "start_time": "2023-07-12T11:50:52.590Z"
   },
   {
    "duration": 215,
    "start_time": "2023-07-12T11:51:16.201Z"
   },
   {
    "duration": 357,
    "start_time": "2023-07-12T11:51:49.134Z"
   },
   {
    "duration": 663,
    "start_time": "2023-07-12T11:51:57.758Z"
   },
   {
    "duration": 3,
    "start_time": "2023-07-12T11:51:58.471Z"
   },
   {
    "duration": 24015,
    "start_time": "2023-07-12T11:51:58.476Z"
   },
   {
    "duration": 5497,
    "start_time": "2023-07-12T11:52:22.493Z"
   },
   {
    "duration": 182,
    "start_time": "2023-07-12T11:52:27.993Z"
   },
   {
    "duration": 3534,
    "start_time": "2023-07-12T11:52:28.177Z"
   },
   {
    "duration": 60,
    "start_time": "2023-07-12T11:52:31.712Z"
   },
   {
    "duration": 317,
    "start_time": "2023-07-12T11:52:31.774Z"
   },
   {
    "duration": 5,
    "start_time": "2023-07-12T11:52:32.093Z"
   },
   {
    "duration": 1518,
    "start_time": "2023-07-12T11:52:32.100Z"
   },
   {
    "duration": 3,
    "start_time": "2023-07-12T11:52:33.620Z"
   },
   {
    "duration": 2284,
    "start_time": "2023-07-12T11:52:33.624Z"
   },
   {
    "duration": 707,
    "start_time": "2023-07-12T11:55:45.869Z"
   },
   {
    "duration": 4,
    "start_time": "2023-07-12T12:02:40.670Z"
   },
   {
    "duration": 7,
    "start_time": "2023-07-12T12:02:40.693Z"
   },
   {
    "duration": 490,
    "start_time": "2023-07-12T12:02:40.701Z"
   },
   {
    "duration": 2880,
    "start_time": "2023-07-12T12:02:41.192Z"
   },
   {
    "duration": 24,
    "start_time": "2023-07-12T12:02:44.074Z"
   },
   {
    "duration": 1401,
    "start_time": "2023-07-12T12:02:44.101Z"
   },
   {
    "duration": 5,
    "start_time": "2023-07-12T12:02:45.504Z"
   },
   {
    "duration": 184,
    "start_time": "2023-07-12T12:02:45.510Z"
   },
   {
    "duration": 76,
    "start_time": "2023-07-12T12:02:45.696Z"
   },
   {
    "duration": 853,
    "start_time": "2023-07-12T12:02:45.774Z"
   },
   {
    "duration": 42,
    "start_time": "2023-07-12T12:02:46.629Z"
   },
   {
    "duration": 904,
    "start_time": "2023-07-12T12:02:46.673Z"
   },
   {
    "duration": 333,
    "start_time": "2023-07-12T12:02:47.579Z"
   },
   {
    "duration": 82,
    "start_time": "2023-07-12T12:02:47.914Z"
   },
   {
    "duration": 91,
    "start_time": "2023-07-12T12:02:47.999Z"
   },
   {
    "duration": 648,
    "start_time": "2023-07-12T12:02:48.092Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-12T12:02:48.742Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-12T12:02:48.743Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-12T12:02:48.744Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-12T12:02:48.745Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-12T12:02:48.746Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-12T12:02:48.748Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-12T12:02:48.749Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-12T12:02:48.750Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-12T12:02:48.751Z"
   },
   {
    "duration": 1266,
    "start_time": "2023-07-12T12:04:16.723Z"
   },
   {
    "duration": 3,
    "start_time": "2023-07-12T12:04:18.619Z"
   },
   {
    "duration": 675,
    "start_time": "2023-07-12T12:04:22.697Z"
   },
   {
    "duration": 547,
    "start_time": "2023-07-12T12:04:27.899Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-12T12:04:28.448Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-12T12:04:28.449Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-12T12:04:28.450Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-12T12:04:28.452Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-12T12:04:28.453Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-12T12:04:28.454Z"
   },
   {
    "duration": 0,
    "start_time": "2023-07-12T12:04:28.455Z"
   },
   {
    "duration": 3052,
    "start_time": "2023-07-12T12:04:41.023Z"
   },
   {
    "duration": 531,
    "start_time": "2023-07-12T12:04:44.077Z"
   },
   {
    "duration": 1730,
    "start_time": "2023-07-12T12:04:44.610Z"
   },
   {
    "duration": 497,
    "start_time": "2023-07-12T12:04:46.341Z"
   },
   {
    "duration": 1672,
    "start_time": "2023-07-12T12:04:46.841Z"
   },
   {
    "duration": 1172,
    "start_time": "2023-07-12T12:04:48.515Z"
   },
   {
    "duration": 594,
    "start_time": "2023-07-12T12:04:49.688Z"
   },
   {
    "duration": 770,
    "start_time": "2023-07-12T12:05:38.343Z"
   },
   {
    "duration": 3,
    "start_time": "2023-07-12T12:05:39.114Z"
   },
   {
    "duration": 24379,
    "start_time": "2023-07-12T12:05:39.118Z"
   },
   {
    "duration": 5407,
    "start_time": "2023-07-12T12:06:03.499Z"
   },
   {
    "duration": 116,
    "start_time": "2023-07-12T12:06:08.971Z"
   },
   {
    "duration": 3382,
    "start_time": "2023-07-12T12:06:09.089Z"
   },
   {
    "duration": 5,
    "start_time": "2023-07-12T12:06:12.473Z"
   },
   {
    "duration": 299,
    "start_time": "2023-07-12T12:06:12.480Z"
   },
   {
    "duration": 4,
    "start_time": "2023-07-12T12:06:12.782Z"
   },
   {
    "duration": 1531,
    "start_time": "2023-07-12T12:06:12.788Z"
   },
   {
    "duration": 4,
    "start_time": "2023-07-12T12:06:14.371Z"
   },
   {
    "duration": 2018,
    "start_time": "2023-07-12T12:06:14.376Z"
   },
   {
    "duration": 778,
    "start_time": "2023-07-12T12:06:16.396Z"
   },
   {
    "duration": 107,
    "start_time": "2023-07-12T12:06:17.177Z"
   },
   {
    "duration": 106,
    "start_time": "2023-07-12T12:06:17.286Z"
   },
   {
    "duration": 1291,
    "start_time": "2023-07-12T12:06:17.394Z"
   },
   {
    "duration": 4,
    "start_time": "2023-07-12T12:06:18.687Z"
   },
   {
    "duration": 822,
    "start_time": "2023-07-12T12:06:18.692Z"
   },
   {
    "duration": 2726,
    "start_time": "2023-07-12T12:06:19.572Z"
   },
   {
    "duration": 614,
    "start_time": "2023-07-12T12:06:22.299Z"
   },
   {
    "duration": 1913,
    "start_time": "2023-07-12T12:06:22.971Z"
   },
   {
    "duration": 586,
    "start_time": "2023-07-12T12:06:24.886Z"
   },
   {
    "duration": 1622,
    "start_time": "2023-07-12T12:06:25.474Z"
   },
   {
    "duration": 1436,
    "start_time": "2023-07-12T12:06:27.098Z"
   },
   {
    "duration": 158,
    "start_time": "2023-07-12T12:06:28.536Z"
   }
  ],
  "colab": {
   "name": "PySparkSprint.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
